the "Retry-After=[]" in this log is now insistent and sleep_amount may not be a meaningful number if retry_after failed to be parsed into a datetime object

sorry, I do not get what [] you have in mind since it can't be a list... please rereview when I push changes -- may be the issue is addressed.

I think I meant that you have numerous logs with inconsistent labelling with "Retry-After=". Some of them you set it to the header value of the "Retry-After" (understandable), others you set it to sleep_amount which can originate from, but not exactly, the passed in header value for "Retry-After" and can default to -1.0.

please check if it is all consistent now -- seems to me, always seems to point to retry_after.

The Retry-After=[]s in log messages, where [] is used to indicate an expression of %d or %s, are consistent within the get_retry_after(), but this last one is still inconsistent with the ones in get_retry_after(). This last one prints the int value of sleep_amount, the result of get_retry_after(), and the others print whatever the value of the HTTP header Retry-After which can be a string of a specific date.

I think we can just remove the use of Retry-After= here. There is plenty of logged messages within get_retry_after()

I actually think this log statement should be removed entirely. Because of the containing if-statement, this log statement doesn't always get executed for all responses that have a Retry-After header. However, at the same time, I think the original intent is to execute this log statement for all responses with a Retry-After header.

We are better off if we just replace the entire if statement:

        if (sleep_amount := get_retry_after(exc.response)) is not None:
            lgr.debug(
                "%s - download failed due to response %d with "
                "Retry-After=%d: %s, will sleep and retry",
                path,
                exc.response.status_code,
                sleep_amount,
                exc,
            )

with

sleep_amount = get_retry_after(exc.response)

and unindent the next log statement as I recommended below.

thanks! done that, pushed

You may also want to do "Retry-After=%s" instead of "Retry-After=%r" if you want the HTTP header appears exactly the same as they are received, without the quotes.

The Retry-After=[]s in log messages, where [] is used to indicate an expression of %d or %s, are consistent within the get_retry_after(), but this last one is still inconsistent with the ones in get_retry_after(). This last one prints the int value of sleep_amount, the result of get_retry_after(), and the others print whatever the value of the HTTP header Retry-After which can be a string of a specific date.

I think we can just remove the use of Retry-After= here. There is plenty of logged messages within get_retry_after()

I think just unindent these lines, and get the log statement out of the if-statement and remove the last log statement at line 1126. See the comments for the last log statement for more details.

ok, makes sense, dedenting the log so we always log how long we are to sleep, not only if random amount due to no Retry-After. I did remove that other log

I actually think this log statement should be removed entirely. Because of the containing if-statement, this log statement doesn't always get executed for all responses that have a Retry-After header. However, at the same time, I think the original intent is to execute this log statement for all responses with a Retry-After header.

We are better off if we just replace the entire if statement:

        if (sleep_amount := get_retry_after(exc.response)) is not None:
            lgr.debug(
                "%s - download failed due to response %d with "
                "Retry-After=%d: %s, will sleep and retry",
                path,
                exc.response.status_code,
                sleep_amount,
                exc,
            )

with

sleep_amount = get_retry_after(exc.response)

and unindent the next log statement as I recommended below.